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Abstract :   This  paper  describes  a  computational  comparison  of  value  iteration 
algorithms  for  discounted  Markov  decision  processes. 


1.   INTRODUCTION 

This  note  describes  the  results  of  a  computational  comparison  of  value 

iteration  algorithms  suggested  for  solving  finite  state  discounted  Markov 

decision  processes.   Such  a  process  visits  a  set  of  states   S  =  {1,2,...M}  . 

When  it  is  in  state   i  ,  one  can  choose  an  action  k   from  the  finite 

action  set   K.  ,  and  then  receive  an  immediate  reward   r.   and  with  prob- 
i  1 

ability   p..   the  process  will  be  in  state   j   at  the  next  period.   The 
object  is  to  maximize,   v(i)  ,  the  maximum  discounted  reward  over  an  in- 
finite horizon  starting  in  state   i  ,  where   3   is  the  discount  factor.   It 
is  well  known  [1]  that  v(i)   satisfies  the  optimality  equation 

f  k      M   k      ] 
v(i)  =  max  ^r  +  3  J   p   v(j)  J-  ( 

k£K.  [  x     j=l  1J     J  KL'L) 


We  record  the  time  for  value  iteration  algorithms  to  obtain  e-optimal 
solutions,   v   ,  to  (1.1),  (i.e.  jv  -  vl   <  e  ,  where   Ivl   =  maxlv(i)l) 

n  n      °°  '   '  oo  * 

i 
on  randomly  generated  problems.   We  look  at  three  classes  of  fifteen  prob- 
lems each  with   3=9  and  e  =  .0001,  where  v(i)  ~  2,000.   Class  1  prob- 
lems have  100  states  and  between  2  and  7  actions  per  state;  class  2  have 
40  states  and  between  2  and  70  actions  per  state,  whereas  class  3  have 
10  states  and  up  to  500  actions  per  state.   Details  of  how  the  problems 
are  generated  and  computing  facilities  used  are  given  in  [12]. 

In  Section  two  we  describe  the  schemes  examined  and  the  various  bounds 
that  can  be  used  for  stopping  them.   Section  three  concentrates  on  one 
scheme  that  did  well  in  the  comparison  -  ordinary  value  iteration  -  and 
looks  an  various  methods  for  eliminating  non— optimal  actions  both 
permanently  and  temporarily. 


2.   SCHEMES  AND  BOUNDS 

The  scheme  usually  described  as  value  iteration  is 


=  max  ^(i)l  =  max  r^  +  B  I     P^.  v(j)V  (2 

keK  [     J   keK  [  L     j=l  1J   n   J 


which  was  discussed  in  [1,3].   In  analogy  with  the  notation  of  linear 
equations  we  call  this  Pre-Jacobi  (PJ).   This  analogy  leads  us  to  think  of 
the  following  alternative  schemes. 

( 


Jacobi  (J) :    v  ,. (l)  =  max 
n+1      ,  ,. 

keK. 

l 


(ri  +  3  2  Pii  Vj))/(1~3pii^  (2, 


k        Kl    k 

Pre-Gauss-Seidel  (PGS) :   v  .. (i)  =  max 


(i)  =  max  jr.  +  B  J     P   v   (J) 

keK. [  3=1 


M   k 


+  B  Z  Pij  V*> 


(2. 


J=i 


k     i_1  k 
Gauss-Seidel  (GS) :       vn+l(i)  =  (AGSVn} (i)  =  maX  \ {ri  +  6   ^   VVn+l(j) 

keK. [  j=l 

M 

k  k 


+  6  p*     v    (j))/(l-Sp^, )  V  (2 

j-i+1     1J      n  X1   j 


Successive   over   Relaxation    (SOR)  :      v      .  (i)    =   io(A0  v   )  (i)    +   (l-w)v    (i) 

n+1         US  n  n 


(PGS)  was  suggested  by  Kushner  [4],  Porteus  [8]  and  Reetz  [10];  (J)  and 
(GS)  is  found  in  [9]  and  SOR  in  [5].   Experiments  with  SOR  suggested  a 
value  of  a)  =  1.28  for  robust  and  speedy  convergence. 


We  require  bounds  on  the  iterates  of  the  scheme  to  ensure  we  stop  when 
v   is  within  a  specified  value  of  the  optimal  v  of  (1.1).   One  can  use 
the  L   norm  bound,  which  says  if    lol    <  a  <  1   for  all  possible 
transition  matrices  in  the  scheme 

v  , .  =  max{sk  +  Qkv  }  (2.6) 

n+1    ,   —       n 

—      k         — 

then   |v  -  vn+1\m  <  a    |vn+1  -  vjj(l-a)    .   For  (PJ)  ,  (J),  (PCS)  and  (GS), 

it  is  trivial  to  show  the  corresponding  Q's  have   L   norm  less  then   3  . 

For  S.O.R.  we  estimate   a  by  |v  ,,  -  v    /|v  -  v   ,1    and  substitute  in 

1  n+1    n'°°  '  n    n-l'» 

(2.6)  to  get  a  heuristic  bound. 

Porteus  [7]  described  tighter  bounds  for  these  schemes,  exploiting  the 

non-negativity  of  the  elements    q..   of   0   in  (2.6).   They  require  cal- 

M  "LJ 

k    V    k 
culation  of   a.  =   2.   1  •  •   f°r  the  maximizing  action  k  at  each  iterate, 


<■■  -     )     q . .  f o] 

1   j-i  « 


and  we  call  these  the  P.C.  bounds  -  (Porteus  with  calculation).   In  [12] 
we  describe  how  to  estimate  the   ot.   initially,  which  avoids  the  calcula- 
tion at  each  step,  but  gives  looser  bounds,  which  we  denote  P.N.C.  - 
(Porteus  no  calculation).   For  the  (PJ)  scheme  we  also  use  the  second  order 
bounds  (S.O.)  described  in  [11],  which  uses  the  last  three  iteration  values 
to  get  a  tighter  lower  bound  than  Porteus' s  bound. 

The  results  are  given  in  the  following  table  where  (Av)  is  the  average 
C.P.U.  time  for  solving  the  fifteen  problems,  S.D.  che  standard  deviation 
of  the  C.P.U.  time,  and   N   the  number  of  problems  that  method  was  quickest 
at  solving. 


TABLE  1 

X  XT' T*T  T /~\  T"\ 

BOUNDS 

class  : 

L  (100  STATE) 

CLASS  2  (40  STATE) 

* 

CLASS  . 

3  (10  STATE) 

METHOD 

■ 

AV. 

S.D. 

N. 

AV. 

S.D. 

N. 

AV. 

S.D. 

N. 

PJ 

PC=PNC 

1.66 

.11 

15 

.79 

.10 

14 

.54 

.07 

4 

SO 

1.69 

.11 

0 

.80 

.11 

1 

.54 

.08 

11 

J 

L 

CO 

11.88 

.59 

0 

14.38 

2.27 

o 

7.66 

1.14    0 

PNC 

10.49 

.63 

0 

11.55 

2.05 

0 

6.90 

1.09    0 

L 

oo 

6.86 

.33 

0 

8.62 

1.34 

0 

5.18 

.82    0 

PGS 

PC 

6.59 

.33 

0 

8.34 

1.32 

0 

5.03 

.81    0 

PNC 

6.60 

.33 

0 

8.40 

1.33 

0 

5.01 

.81    0 

L 

oo 

6.55 

.32 

0 

8.13 

1.41 

0 

3.99 

.61    0 

GS 

PC 

6.32 

.34 

0 

7.77 

1.38 

0 

3.90 

.61 

0 

PNC 

6.25 

.32 

0 

7.70 

1.41 

0 

3.83 

.61 

0 

SOR 

L 

oo 

3.30 

.22 

0 

4.00 

.55 

0 

1.86 

.30 

0 

For  Jacobi,  the  P.C.  bound  is  the  same  as  the  P.N.C.  bound  and  so  the 
latter  must  be  faster  as  it  involves  less  calculation.   It  is  obvious  from 
Table  1  that  P.J.  with  Porteus  bounds  performs  very  well,  and  in  the  next 
section  we  concentrate  on  this  scheme  and  apply  elimination  of  non-optimal 
actions. 


3.   ACTION  ELIMINATION 

MacOueen  [6]  described  how  for  any  bounds  one  can  observe  a  test  to 
identify  actions  that  cannot  optimize  the  right  hand  side  of  (1.1)  and  so 
can  be  permanently  eliminated  from  the  calculation.   Applying  MacQueen's 
bounds  [6]  and  Porteus 's  bound  [7]  for  the  PJ  algorithm  leads  to  the 
following  tests  to  eliminate  action  k.  in  K.   permanently. 


MacQueen        v  (i)  <  v  (i)  +  3(a  -  b  )/(l-B)  (3.1) 

Porteus         vk(i)  <  v^(i)  +  B^a^  -  b^^/Cl-B)  (3.2) 


where  a  =  min(v  (1)  -  v   n(i))  ,      b   =  max(v  (1)  -  v   ,(i))  . 
n     .    n       n-i  n     .    n       n— 1 

i  l 

We  looked  at  four  ways  of  implementing  these  tests. 

Ml.   At   n    iteration,  calculate  and  store  v  (i)   for  each   i  .   Then 

n 

calculate   a   and  b   .   Recalculate  v  (i)   and  use  (3.1)  to  test 
n        n  n 

for  elimination. 

th  k 

M2.   At   n    stage,  calculate  and  store  all  v  (i)  .   Hence  calculate 

n 

v  (i)  ,  a   ,  b    and  test  for  elimination  using  (3.1)  without 
n       n    n 

recalculating  v  (i)  . 

th  k 

PI.   At   n+1    stage,  calculate   v  ,,(i)  ,  starting  with  action  k   that 

n+I 

maximized   v  (i)   at  previous  stage.   Apply  (3.2)  as  soon  as  you  cal- 
n 

culate  each  v    (i)   using  as   d   the  one  that  gives  maximum 

v  ,,(i)   so  far  calculated,  see  [7]. 
n+1 

th  k 

P2.   At   n+1    stage,  calculate  and  store   v   , (i)  .   Then  using 

vn+1(i)  =  vn+1(i)   apply  (3.2). 

As  Table  2  shows  M2  is  far  superior  to  Ml,  but  PI  and  P2  give  similar 
results.   All  three  cut  the  average  time  by  a  half  though. 

Hastings  and  Van  Nunen  [2]  pointed  out  that  one  could  also  eliminate 
actions  temporarily,  i.e.  actions  that  will  not  be  the  optimizing  actions 
at  the  next  iteration  of  the  PJ  algorithm.   This  is  based  on  the  inequality 


vk^  (i)  -  vk.  (i)  >  v  (i)  -  vk(i)  -  3  I      (b  ,.  .  -  a  , .  .)         (3.3) 
n+s       n+s    —  n       n        ,u-         n+i-1    n+i-1 

j  =  l 


If  the  R.H.S.  of  (3.3)  is  positive  k  will  not  optimize  the  n  +  s — 

t*i_ 

iteration,  and  in  that  case,  at  the  n  +  s  +  1 —  iteration  we  need  only 
subtract  another   3(b   ,  -  a   )   from  this  positive  number  to  test  if  k 

LI  i  o        11  >^o 

could  be  optimal.   If  action  k  is  not  eliminated  at  the  n  +  s — 

V 
iteration,   v   (i)  -  v   (i)   is  stored  for  the  test  at  the  next  iteration, 
n+s       n+s 

We  looked  at  four  ways  of  implementing  these  two  elimination  procedures. 
Recall  that  the  n  +  1 —  iteration  consists  of  the  following  sequence  of 
calculations. 


(I)    k       (ID          (HI)  k 

Vbn  *  Vn+l(i)  ^  Vn+l(i)  ►  an+l'bn+l  *  Vn+2(i) 


TEMP  HVN.   Hastings  and  Van  Nunen  [2]  suggested  the  temporary  elimination 
test  be  made  at  (I)  and  if  k  was  not  temporarily  eliminated  then  v   (i) 
was  calculated.   The  permanent  elimination  test  was  made  at  (II)  using 

(3.2)  with  v  . ,  (i)   replaced  by  a  lower  bound  v  (i)  +  3a   .   If  the 

n+i  n        n 

action  is  not  permanently  eliminated,   v  (i)  +  3a  —  v  ,,(i)   (rather  than 

n        n    n+1 

v  ,  ,(i)  -  v  ,,(i))   is  stored  TEMP  +  PI.   Temporary  elimination  occurs  at 
n+1       n+I 

(I)  and  permanent  elimination  at  (II)  using  PI  . 

TEMP  +  P2.   Again  this  has  temporary  elimination  at  (I)  and  permanent 
elimination  at  (III)  using  P2  . 

TEMP  +  M2.   Temporary  elimination  occurs  at  (I)  and  in  this  case  v    (i) 
was  stored  until  (IV)  and  then  the   M2  technique  used.   If  action  k  was 

not  eliminated  v  ,  ,(i)  -  v  , ,  (i)   was  stored  for  the  temporary  elimination 

n+i       n+1 

test  of  the  next  iteration,  which  followed  immediately. 

In  this  case  when  permanent  and  temporary  elimination  are  done  at 
the  same  stage,  it  is  obvious  that  any  action  which  is  permanently  eliminated 
would  also  be  temporarily  eliminated. 


This  leads  us  to  ask  is  it  worth  permanently  eliminating,  so 
flLY  Perform  the  temporary  elimination  test  at  (I).   If  action  k  is  not 
eliminated  v   - (i)   is  stored  through  state  (II)  and  changed  to 
v  +1(i)  -  vn+;i/i)   at  sta8e  (HI)  for  the  next  temporary  elimination  test 
at  (IV). 

Table  2  describes  the  results  and  shows  that  temporary  elimination 
further  cuts  the  time  by  25%,  and  that  pure  temporary  elimination  might 
be  particularly  good  on  large  scale  problems. 


TABLE  2 


METHOD 

class  : 

L  (100  STATE) 

class  ; 

I    (40  STATE) 

CLASS  . 

3  (10  STATE) 

AV. 

S.D. 

N. 

AV. 

S.D. 

N. 

AV. 

S.D. 

N. 

Ml 

1.51 

.09 

0 

0.67 

.08 

0 

0.42 

.06    0 

| 

M2 

0.81 

.05 

15 

0.36 

.04 

15 

0.25 

.03   15 

PI 

0.87 

.05 

0 

0.43 

.05 

0 

0.26 

.03    0 

P2 

0.88 

.05 

0 

0.45 

.05 

0 

0.28 

.04  ;     o 

TEMP  HVN 

0.62 

.03 

0 

0.27 

0.03 

0 

0.21 

.03  |   0 

TEMP  +  PI 

0.60 

.04 

0 

0.25 

.02 

3 

0.20 

.03 

11 

TEMP  +  P2 

0.58 

.03 

0 

0.26 

.02 

0 

0.23 

.03 

0 

TEMP  +  M2 

0.59 

.04 

0 

0.22 

.04 

2 

0.21 

.03 

4 

TEMP  ONLY 

0.55 

.03 

15 

0.22 

.04 

10 

0.21 

.03 

0 

Our  object  has  not  been  to  obtain  a  best  buy,  but  to  give  some  idea  of 
the  merits  of  the  various  schemes,  bounds  and  improvements.   Obviously  for 
more  structured  problems,  algorithms  which  exploit  the  structure  will  be 
at  an  advantage. 
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